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Abstract 

The researchers explored whether implementation of a system¬ 
atic outcomes-based assessment process is necessary for demon¬ 
strating quality in service learning programs at a two-year and a 
four-year institution. The findings revealed that Western Com¬ 
munity College and the University of the Coast maintained 
quality service-learning programs, which met their established 
learning goals and reflected the community’s needs, by incor¬ 
porating numerous components of a systematic outcomes-based 
assessment process, even though this process had not been es¬ 
tablished. 


CAN INSTITUTIONS HAVE QUALITY 
PROGRAMMING WITHOUT UTILIZING 
A SYSTEMATIC OUTCOMES-BASED 
ASSESSMENT PROCESS? 

For many students, service learning not only expands their educational horizons, 
but also makes them more aware and compassionate of those who live in communities 
very different from their own. Yet, despite the expressed benefits of service learning 
and the increasing numbers of institutions offering service-learning programs, it is not 
known whether they are truly successful unless evidence of their success is provided. 
The use of outcomes-based assessment is one of the processes that generates evidence 
of program effectiveness. While scholars have written about the ways in which effective 
outcomes-based assessment must be implemented (Bresciani, 2006; Bresciani, Moore- 
Gardner, & Hickmott, 2009; Maki, 2004; Palomba & Banta, 1999; Suskie, 2004; 
Upcraft & Schuh, 1996), few studies have been performed to identify whether all the 
components of effective outcomes-based assessment must be present in order for quality 
programs to be identified. Thus, the purpose of this cross-case comparative study was 
to find out whether implementation of a systematic outcomes-based assessment system 
is necessary for demonstrating quality in service learning at a community college and a 
private four-year institution. 

Review of Literature 

Bresciani (2006) noted that outcomes-based assessment provides vital 
information to enhance learning through “systematic evaluation of student learning and 
development” (p. 1). In addition, according to Bresciani et al. (2009), “Outcomes-based 
assessment is intended to inform decision-making processes”(p. 14) and “goes one step 
beyond typical evaluation” (p. 16). Bresciani noted that a strategy is devised to accomplish 
the intended goals after determining what the curriculum is to achieve “in regard to its 
services, research, student learning, and faculty/staff development programs” (p. 14). A 
systematic method of evaluation, outcomes-based assessment is a continuous process, 
which recurs on a timeline in order to ascertain “whether the program improvements 
contribute to the intended outcomes” (Bresciani, 2006, p. 14). 

As noted in the literature, there are several integral components that make up 
a quality outcomes-based assessment process (Bresciani, 2006; Bresciani, Zelna, & 
Anderson, 2004; Palomba & Banta, 1999; Suskie, 2004). These components should 
include: (a) defining mission statements, (b) defining goals, (c) distinguishing between 
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institutional, division, and programmatic learning outcomes, (d) mapping learning 
outcomes, (e) planning the delivery of your outcomes, (f) choosing a method and criteria 
to assess, (g) reporting and interpreting results, (h) using results to make decisions and 
recommendations, and (i) establishing a timeline for implementation and improvement 
(Bresciani et al., 2009). 


The researchers were interested in learning from the students, faculty, staff, and 
community partners of Western Community College and the University of the Coast 
whether these two institutions incorporated a systematic outcomes-based assessment 
process when evaluating the effectiveness of their service-learning courses. However, if the 
institutions utilized outcomes-based assessment, how did they handle this process? 


From this study, the researchers were able to determine that Western Community 
College and the University of the Coast evaluated the effectiveness of their service-learning 
programs by (a) the success of their collaborative partnerships, (b) the successful connection 
of their students’ curricular and co-curricular experiences, (c) the availability of reflection 
opportunities, (d) the depth of the students’ feedback and assessment opportunities, and (e) 
the sustainability of the program. Although there were many similarities between Western 
Community College and the University of the Coast, only the University of the Coast 
incorporated a one-unit for credit leadership course, which was designed to train selected 
service-learning student leaders to work as site coordinators. Site coordinators serve as 
intermediaries between the service-learning students, faculty, community partners and the 
Community Service-Learning Center to make sure that the academic aspects of the course 
are being met along with the needs of the partners. The following study illustrates the 
process used to discover how and which of these portions of the systematic outcomes-based 
process are present. 


“Outcomes-based 
assessment is 
intended to inform 
decision- making 
processes and goes 
one step beyond 
typical evaluation.” 


Methodology 

Research Design 

This cross-case comparative study explored through a constructive lens whether 
implementation of a systematic outcomes-based assessment process is necessary to 
determine whether quality of service-learning programs are present in a four-year private 
institution and a public community college, referred to as the University of the Coast and 
Western Community College. The researchers selected a multiple case study as the specific 
methodology type since it works very well for gathering information to explore the factors 
that may lead to sustainable, high-quality service learning programs and since this model is 
very appropriate “whenever it is human inquiry that is being considered” (Cuba & Lincoln, 
1989, p. 82). The researchers also chose the constructive method since its flexibility enabled 
them to pursue avenues of investigation that may arise during the investigation process, 
making this paradigm open to finding out new information and substantiating it (Guba & 
Lincoln, 1989). The researchers defined quality based upon an extensive literature review 
of the characteristics of quality service-learning programs and by the research process that 
uncovered components of a systematic outcomes-based assessment. 

The researchers utilized multiple sources of data collection, including one-on-one 
interviews and focus groups with students, faculty, staff members, and community partners 
and on-site observations as well as document analysis of (a) students’ journals from class, 
(b) faculty or staff reports, (c) student assessment, (d) faculty and staff assessments, and (e) 
partnership assessments. The researchers also reviewed the assessment plans of students, 
faculty and or staff members, and community partners. 

Site Selection and Sampling Method 

The researchers selected Western Community College’s Service-Learning Program 
and the University of the Coast’s Community Service-Learning Center for the study since 
these particular institutions have long established service-learning programs with strong 
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“In our opinion, 
Western Community 
College and University 
of the Coast are 
successful since both 
institutions use a 
variety of ways to 
determine learning 
outcomes for their 
service-learning 
courses.” 


ties to their respective communities. In addition, these two institutions are very different 
from each other. For example, Western Community College is large, two-year public 
educational institution relying on state funding and the University of the Coast is a 
private, four-year church-based educational institution with an endowment of hundreds 
of millions of dollars. 

There were 26 participants in the study, five of whom were males. In addition 
to one-on-one interviews with 18 persons including (a) five students, (b) six faculty 
members, (c) three staff members, and (d) four community partners, there were two focus 
groups with a total of eight students. The researchers utilized purposeful sampling and 
asked the service-learning administrators at both institutions to identify faculty or staff 
members for one-on-one interviews, who had different areas of service-learning expertise 
and varying degrees of experience in service learning. Based on the recommendations of 
the service-learning coordinators, the researchers interviewed community partners from 

(a) both small and large non-profit organizations, (b) long established and newly formed 
non-profit organizations, (c) those working with different age groups, different socio¬ 
economic groups, and different ethnic groups. 

In framing the interview protocols for one-on-one interviews and focus groups, 
the researchers selected a set of specific questions derived from a review of literature, 
which added focus and meaning to the research question. These particular questions 
provided insight into what factors the participants believed were essential components 
of quality service-learning programs. Another integral component of this cross-case 
comparative study were observations at partner sites and a document analysis of official 
documents, such as strategic plans, assessment plans, student evaluations, reflections, and 
mission statements. These public documents not only enriched the study and helped the 
researchers determine if these two institutions of higher education had the characteristics 
of quality service-learning programs, but also supplemented the other methods of the 
study, providing more depth to the researchers’ findings. 

Data Analysis 

Using a “process of inductive reasoning, thinking, and theorizing” (Taylor & 
Bogdan, 1998, p. 140), the researchers analyzed recorded transcripts of the one-on- 
one interviews and focus groups. From the transcripts, the researchers captured from 
students, faculty, staff members, and partners their feelings about their service-learning 
programs including (a) their reactions to their service-learning experience, (b) their 
definition of sustainability, (c) their interactions with all of the entities that comprise the 
program, and (d) any issues of concern. 

Findings and Discussion 

While determining whether each institution has a systematic outcomes-based 
assessment process in place, and if not, identifying what components of a systematic 
outcomes-based assessment process exist, the researchers learned there is close 
communication among all of the entities of both service-learning programs. Although 
the University of the Coast’s Community Service-Learning Center and Western 
Community College’s Service-Learning Program do not have a systematic outcomes- 
based assessment process, both programs have numerous components of the process 
in place, such as (a) incorporating feedback, including surveys and evaluations from 
students, faculty, and community partners as part of an internal assessment process to 
make necessary changes that have improved their respective service-learning programs; 

(b) having the service-learning administrators interface with faculty on an individual 
basis to review the service-learning component for their courses and devise learning 
outcomes; (c) providing numerous orientation and training opportunities, including 
workshops to assist faculty in developing learning outcomes; and (d) encouraging close 
communication, enabling the entities in the partnership to interface and share their ideas 
and concerns. Both institutions communicated “a shared purpose and vision.. .defining 
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goals” (Bresciani et al., 2009, pp. 36-37), which should be present in an outcomes-based 
assessment. 

A formalized feedback and assessment process is an important factor leading to 
the success of both service-learning programs. Both the service-learning coordinator at 
Western Community College and the director and assistant director of the University of 
the Coast’s Community Service-Learning Center place great emphasis on both feedback 
and assessment by maintaining close communication with all entities providing (a) guidance 
about the purpose, goals, and logistics of service-learning programs, (b) information about 
the community partners and how their mission may complement the academic aspects of 
service-learning courses, and (c) information about the service-learning sites to assist faculty 
in designing and implementing programs to achieve the expected learning outcomes. This 
information not only educates students, faculty, and partners, but also enables them to 
solicit feedback, strengthening the assessment process. 

Although neither the University of the Coast nor Western Community College 
utilize a systematic outcomes-based process, faculty members meet with service-learning 
program administrators to discuss learning outcomes. According to Bresciani (2006), in 
an outcomes-based assessment, it is important to “engage in these conversations faculty 
and staff who are well respected in their disciplines and in their research” to lessen .the 
common misconceptions that outcomes-based assessment is just another fad or a process 
built by administrators to ‘check up’ on the work of the faculty” (p. 19). Bresciani also 
noted that the information compiled through this process promotes discussions about 
responsibility and may lead to “opportunities for improvement” (p. 15). 

From the interviews and document analysis, the researchers determined that 
Western Community College established several outcomes in its Unit Assessment 
Report (2008-2009, pp. 1-4) that included (a) increasing the service-learning skills of its 
student advocates by providing leadership training to plan a community service project, 
(b) advancing critical and reflective thinking skills to encourage connections between 
curricular and co-curricular experiences, and (c) increasing student knowledge about the 
organizations which are their community service-learning partners. However, the Unit 
Assessment Report tracks only a few objectives and does not reflect the work of the entire 
program. According to the University of the Coast Strategic Plan, “Our key success factors 
are our outcome measures of success. They measure how successfully the University of the 
Coast Community Service-Learning Center is achieving our Mission, Vision 2012, and 
our Core Values on a year-by-year basis” (2008-2012, p. 8). 

In our opinion, Western Community College and the University of the Coast are 
successful since both institutions use a variety of ways to determine learning outcomes 
for their service-learning courses. Neither institution utilizes a systematic outcomes based 
assessment that provides integration between course outcomes and program outcomes, the 
purpose of which is to provide “quality assurance and external accountability in higher 
education” (Bresciani, 2006, p. 13). From the comments of the participants in the study, the 
researchers learned that the instructors at Western Community College and the University 
of the Coast make their students aware of the expected learning outcomes from their 
courses. For instance, in an ongoing systematic manner, both institutions have (a) devised 
the learning outcomes for their courses, (b) gathered feedback, (c) assessed it, (d) made 
necessary changes, and (e) evaluated the results of the changes to determine if students 
are achieving the expected learning outcomes purposefully established, a process that has 
characteristics of an outcomes-based assessment (Bresciani, 2006). 

The document analysis not only provided insight into the assessment procedures 
at the University of the Coast and Western Community College, but also reinforced the 
findings from the interviews and focus groups. For example, Western Community College’s 
Unit Assessment Report 2008 went into great detail regarding Western Community 
Colleges outcomes such as increasing the service-learning skills of its student advocates by 
• «. 
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providing leadership training to plan a community service project and increasing student 
knowledge about the organizations which are community service-learning partners. The 
University of the Coast’s service-learning rubric was very specific and provides guidelines 
and structure. It can be an excellent tool to assess academic achievement of the curricular 
and co-curricular experiences and may provide as much or more structure for systematic 
assessment than a formalized plan would provide. In addition, in the University of the 
Coast Community Service-Learning Center Strategic plan (2008-2012), the Community 
Service-Learning Center placed a great deal of emphasis on the assessment and evaluation 
of its service-learning programs and incorporated these categories into the document. 

Although Western Community College is not utilizing an ongoing systematic 
outcomes-based assessment at this time, the coordinator is working with the faculty and 
administrators to establish overall learning outcomes of the service-learning program 
and not just the learning outcomes for particular courses. She is putting into place “a 
systematic process of evaluation . . . repeated at a later date to determine whether the 
program improvements contribute to the intended learning outcomes” to incorporate 
components of an ongoing systematic outcomes-based assessment (Bresciani, 2006, p. 
14). According to McEwen (1996), “Carefully designed service-learning experiences can 
lead to profound learning and development outcomes for students, the primary reason 
that institutions of higher education engage in service-learning” (p. 53). Matt, assistant 
director of the service-learning programs at the University of the Coast, also discussed 
the importance of establishing learning objectives in designing programs and “getting 
a buy-in” from all of the stakeholders. “Getting their [stakeholders’] input and seeing 
how they think the program should be run . . . helps guide you,” Matt continued. Even 
though the University of the Coast does not have a systematic on-going outcomes-based 
assessment process, they are devising learning outcomes not only for individual courses, 
but also for their overall service-learning program. 

The close interaction and ongoing quality checks between both the service¬ 
learning administrators at University of the Coast and at Western Community College 
and their respective stakeholders is instrumental in achieving the intended outcomes of 
their service-learning programs. Constant communication between all of the entities 
is essential to preserve quality, especially when there are changes in the staffing at a 
community partner’s service-learning site. Based on the study, an important component 
of Western Community College and the University of the Coast’s success is the fact that 
both institutions have strong and stable service-learning departments where their staff 
members maintain close ties with all of the stakeholders. 

In addition, both institutions’ service-learning departments work directly with 
faculty, helping them to assess learning outcomes for the course on an individual basis, 
but the learning outcomes are not integrated with program outcomes. However, the 
importance of utilizing a formalized systematic on-going outcomes-based assessment 
process cannot be overlooked. For example, if there were changes in the hierarchy of the 
service-learning programs at either institution and a new coordinator or director replaced 
Michelle or Wendy, gaps in leadership may develop, and their successful service-learning 
programs may suffer. 

The University of the Coast and Western Community College’s service¬ 
learning programs succeeded without a systematic ongoing outcomes-based assessment 
review due to the fact that they implemented the characteristics of a quality service¬ 
learning program, including informally and continuously soliciting feedback and making 
evaluations. However, other two-year and four-year institutions may not experience 
the same results without a systematic outcomes-based assessment process if they do not 
incorporate the same quality characteristics as the University of the Coast and Western 
Community College. 

Even though an outcomes-based assessment process does not exist at the 
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University of the Coast’s Community Service-Learning Center and Western Community 
College’s Service-Learning Program, based on this study, both institutions appear to have 
quality service-learning programs as noted by (a) their sustainability; (b) the satisfaction of 
their students, faculty, and community partners; (c) the agreement on the intended learning 
goals and how the program meets the needs of the community; (d) and the learning that 
takes place, which is reflected in the feedback and reflection activities. From the study, 
the researchers determined that both institutions (a) have clearly defined their mission 
statements through an excellent communication network, (b) have carefully identified the 
goals of their individual service-learning programs even though this identification is not 
always a collaborative effort between the faculty and service-learning staff, but at a later 
time is mutually discussed and understood, (c) selected appropriate assessment methods 
through a variety of different processes appropriate for the particular program, (d) report 
and interpret results to all of the partners, and (e) use the results to make decisions and 
recommendations for program improvement. 

While faculty members and service-learning administrators at Western Community 
College and the University of the Coast devise learning outcomes and are working towards 
distinguishing between and integrating the different types of learning outcomes, the 
process is still under development at both institutions. The institutions’ missing pieces are 
(a) not incorporating a segmented process for the developing the achievement of learning 
outcomes, (b) the lack of integration between service-learning administrators and faculty in 
identifying whether the learning outcomes have been achieved, (c) the lack of a systematic 
process for conducting assessments, (d) not systematically identifying, the results of the 
learning outcomes, (e) not consistently utilizing the results of the assessment for program 
improvement or recommendations, and (f) not consistently repeating the evaluation 
process to measure whether the improvements or recommendations have resulted in the 
achievement of the learning outcomes. 

Conclusion and Implications for Practice 

The findings of this study reveal that while a systematic outcomes-based assessment 
process did not exist at the University of the Coast and Western Community College, 
there was evidence of quality in both programs that were studied since both institutions 
demonstrated several effective assessment practices. For example, they placed great 
emphasis on feedback and assessment between their various constituent groups. Western 
Community College has also increased the frequency of its program reviews and has 
incorporated learning outcomes into the process, and the University of the Coast has 
created a Service-Learning Student Learning Rubric. In addition to surveys and the rubric, 
some professors teaching service-learning courses have students write literacy case studies. 
For example, according to third-year student, Alice, literacy case studies not only help to 
evaluate the progress service-learning students made tutoring youngsters, but also enable 
them to make suggestions for improvement. The University of the Coast uses both formal 
assessments with a paper trail and informal assessments, and according to Wendy, director 
of the University’s Community Service-Learning Center, “With the partners, we moved 
from formal to more checking in at the beginning and end of each semester with personal 
conversations.” Feedback has always been an essential part of the University’s assessment 
process and, according to the University of the Coast Community Service-Learning 
Center Annual Report (2005-2006), great emphasis is placed on student feedback to 
determine what students are learning from their community experiences. Feedback from 
this objective came from an evaluation by 256 students replying to 15 statements answered 
on a Likert scale; “97% of the students responded favorably that the service experience 
helped them better understand their organization” (Western Community College Unit 
Assessment Report, 2008-2009, p. 4). 


“...unless an 
institution has a 
commitment from 
its leadership to 
continue a program 
regardless of the 
quality of data they 
are able to produce 
in a systematic 
way, it may be 
wise to engage 
in a systematic 
outcomes-based 
assessment process 
or at the very least, 
engage researchers 
to conduct a 
thorough case study 
so that quality data 
and evidence of 
decision-making 
can emerge and be 
documented.” 
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The researchers recommend that both institutions continue to expand on these 
assessment practices, such as the feedback and reflection activities that they have in place. 
However, the level of quality could only be identified because the researchers conducted a 
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thorough cross-case comparative study in order to find the data that revealed the quality 
of the programs. Thus, unless an institution has a commitment from its leadership 
to continue a program regardless of the quality of data they are able to produce in a 
systematic way, it may be wise to engage in a systematic outcomes-based assessment 
process or at the very least, engage researchers to conduct a thorough case study so that 
quality data and evidence of decision-making can emerge and be documented. 

While it may not be necessary for both institutions to incorporate an ongoing 
systematic outcomes-based assessment process at this time because of their institutional 
leadership commitment to continue service-learning, the researchers believe it would 
be helpful for them to do so given the scope of their service-learning programs and the 
continued expansion of their outreach student and community outreach efforts. After 
a systematic outcomes-based assessment process is developed and in place, it may take 
less time to generate accurate data to determine whether there is a quality program, 
thus, saving time later in replicating a similar study to the one conducted at Western 
Community College and the University of the Coast. 
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